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Foreword 


Research  discussed  in  this  report  is  part  of  an  exploratory  development  project 
entitled  “Information  Delivery  System  Design  for  Personnel  Force  Management” 
(Program  Element  0602233N,  Project  RM33M20).  The  report  describes  a  technique  for 
quantifying  the  relationship  between  a  graphical  metaphor  of  a  software  application  and 
the  mental  model  of  the  user.  The  technique,  based  on  similarity  ratings  between 
properties  of  a  metaphor  and  the  user’s  concept  of  the  software,  was  developed  by 
researchers  at  the  Navy  Personnel  Research  and  Development  Center  with  assistance 
from  BBN  Corporation,  Systems  and  Technologies  Division  (Contract  No.  N0244-95-D- 
0281).  The  visual  metaphor  in  this  study  was  a  hydraulic  system  that  represented  the 
process  of  “A”  School  planning.  This  metaphor  was  also  used  in  a  previous  study 
conducted  during  this  project  (MacMillian  Freeman,  Tatum,  &  Ropp,  1998).  The  study 
suggests  that  software  designers  should  align  their  graphical  metaphors  with  the  mental 
models  of  the  users,  and  proposes  a  useful  technique  to  accomplish  this  objective. 


W.  M.  KEENEY  MURRAY  W.  ROWE 

Commander,  U.  S.  Navy  Technical  Director 

Commanding  Officer 
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Summary 


Problem 

Designers  of  software  applications  and  systems  sometimes  attempt  to  increase  user- 
friendliness  by  developing  metaphors.  For  example,  Apple  computers  use  a  desktop 
metaphor  with  their  operating  system,  and  Microsoft  uses  a  typewriter  metaphor  with 
their  popular  word  processor.  But,  do  these  metaphors  match  the  mental  models  that 
users  have  for  these  same  applications?  In  other  words,  does  the  metaphor  correspond  to 
the  way  the  user  conceives  of  the  software  designer's  application? 

Objective 

This  research  attempts  to  develop  a  quantitative  technique  for  measuring  the 
congruence  between  a  graphical  metaphor  and  the  user’s  mental  model.  If  such  a 
technique  is  feasible,  then  software  designers  could  be  more  confident  that  the  way  they 
think  about  the  application  matches  the  way  the  user  thinks  about  the  application.  Such  a 
technique  would  be  valuable  for  both  the  designer  and  the  user.  Designers  could  create 
more  user-friendly  software.  Users  would  find  the  software  easier  to  learn  and  operate. 

Approach 

A  hydraulic  system — a  system  of  tanks,  pipes,  and  valves — was  used  as  a  visual 
metaphor  for  a  Navy  school-planning  process.  (The  visual  metaphor  was  implemented  in  a 
software  program  known  as  SKIPPER).  The  hydraulic  system  was  designed  as  a 
reasonable  approximation  of  the  users’  mental  model  of  the  process.  The  congruence 
between  the  hydraulic  metaphor  and  the  users’  presumed  mental  model  was  tested  using  a 
“Metaphor  Rating  Sheet.”  The  sheet  contained  different  concepts  of  hydraulics  and  school 
planning  as  rows  (e.g.,  tanks,  pipes,  school  capacity,  school  inputs),  and  functional 
properties  of  hydraulic  and  school  planning  as  columns  (e.g.,  holding  capacity,  flow  rate). 
The  user  was  required  to  rate  the  correspondence  between  columns  and  rows  on  a  5-point 
rating  scale. 

Results 

Analysis  of  the  ratings  showed  that  there  was  a  close  parallel  between  the  different 
hydraulic  and  school-planning  concepts  and  their  functional  properties.  For  example,  a 
faucet  (a  component  of  a  hydraulic  system)  and  the  concept  of  loss  rates  (a  school 
planning  concept)  both  share  the  functional  property  of  “exit  out  of  a  system.”  The  pattern 
of  the  findings  revealed  not  only  that  the  hydraulic  metaphor  was  appropriate  for  the  task, 
but  that  the  rating  technique  could  reliably  measure  the  congruence  between  visual 
metaphor  and  the  user’s  mental  model. 

Conclusions 

The  methods  developed  in  this  study  can  be  used  to  measure  the  “fit”  between  a 
designer’s  visual  metaphor  and  the  user’s  mental  model.  The  methods  can  also  be  adapted 
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to  compare  different  metaphors  and  determine  their  relative  congruence  with  the  users’ 
conceptual  model.  However,  the  methods  do  not  tell  the  designer  whether  employing  a 
visual  metaphor  will  facilitate  the  use  of  any  particular  application.  There  may  be  a  strong 
congruence  between  the  metaphor  and  a  user's  mental  model,  but  this  is  still  no  guarantee 
that  the  metaphor  will  help  the  user. 

Recommendations 

1.  Software  and  system  designers  should  use  the  technique  reported  here  (or  some 
variation  on  this  method)  to  determine  whether  the  visual  metaphors  they  use  are 
congruent  with  the  mental  model  of  the  user.  If  there  is  low  congruence,  the  designers 
should  consider  different  metaphors.  If  there  is  high  congruence  (as  was  shown  in  this 
study)  the  designers  can  be  confident  that  their  metaphors  may  help  users  understand  and 
operate  applications  and  systems. 

2.  Future  work  should  focus  on  refining  this  technique  and  developing  variations  that 
allow  comparisons  between  alternative  metaphors. 
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Introduction 


Use  of  the  term  “mental  model”  has  become  widespread  in  discussions  of  how  people 
understand  and  interact  with  complex  systems.  Mental  models  have  been  proposed  as  a 
mechanism  by  which  users  of  a  computer  system  acquire  and  apply  knowledge  about 
how  to  interact  with  the  system  (Moray,  1996;  Carroll  &  Olson,  1990;  Preece  et  al., 
1994).  Two  types  of  mental  models  may  be  useful  in  interacting  with  a  complex  system: 
structural  or  conceptual  models  that  specify  “how  it  works”  and  functional  models  that 
specify  “how  to  do  it.”  Conceptual  models  are  useful  in  understanding  complex 
interactions  and  making  predictions,  while  functional  models  link  actions  to  outcomes, 
but  need  not  explicitly  represent  the  causal  mechanisms  behind  the  link  (Preece  et  al., 
1994). 

Mental  models  are  often  described  in  terms  of  imagery  (Johnson-Laird,  1983),  and 
there  inconsiderable  evidence  that  the  formation  and  use  of  mental  models  draws  on 
components  of  the  human  visual  system  (Kosslyn,  1989).  Because  mental  models  are 
believed  to  have  a  strong  visual  component,  it  is  only  natural  to  think  about  using  visual 
images  or  pictorial  metaphors  to  represent  or  support  them  in  computer  displays.  It  has 
often  been  suggested  that  metaphors  are  useful  in  the  design  of  user  interfaces,  and,  to 
the  extent  that  they  are  congruent  with  underlying  conceptual  models  of  a  system,  that 
such  metaphors  can  be  helpful  in  learning  to  use  a  complex  system  (Carroll,  Mack,  & 
Kellogg,  1990).  Pictorial  metaphors  are  useful  because  of  parallelism  between  the 
pictorial  representation  and  the  thing  being  represented.  This  parallelism  implies  that  the 
representation  and  the  thing  are  from  different  domains  (or  they  could  not  be  parallel) 
and  that  there  is  some  resemblance  between  them,  that  is,  they  have  particular  aspects  or 
properties  that  are  parallel  (Dent-Read,  Klein,  &  Eggleston,  1994). 

Human  factors  scientists  trying  to  evaluate  the  usefulness  of  visual  metaphors  in 
display  design  face  a  complex  measurement  problem  as  illustrated  in  Figure  1.  It  is 
relatively  straightforward  to  measure  whether  the  presence  of  a  visual  metaphor  in  a 
display  design  improves  the  users’  understanding  of  causal  relationships  or  the  users’ 
performance  in  accomplishing  tasks.  What  is  lacking  is  a  method  for  assessing  why  a 
particular  visual  metaphor  is  or  is  not  useful.  Metaphors  may  fail  for  at  least  three 
reasons.  First,  pictorial  metaphors,  in  general,  may  be  a  poor  method  for  improving 
performance  or  increasing  understanding  for  a  specific  application.  Second,  the  metaphor 
may  be  poorly  implemented  in  the  specific  interface,  e.g.,  it  may  be  difficult  to  access  or 
may  not  engage  the  attention  of  the  user.  Third,  metaphors  may  fail  because  the  specific 
metaphor  used  is  a  poor  representation,  i.e.,  it  is  not  congruent  with  the  user’s  mental 
model.  In  order  to  separate  the  third  cause  of  failure  from  the  first  two,  we  need  to 
measure  the  congruence  between  the  properties  of  the  visual  metaphor  and  the  properties 
of  the  user’s  mental  model — we  need  a  method  for  comparing  two  intangibles. 
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Figure  1.  Measurement  challenge  for  evaluating  visual  metaphors 
in  display  design. 


Method 

We  developed  a  measurement  method  for  assessing  the  congruence  between  the 
perceived  properties  of  a  visual  metaphor  for  a  complex  system  and  the  properties  of  the 
users’  conceptual  model  of  that  system.  This  method  was  developed  as  part  of  an  overall 
evaluation  of  a  “User  Coach,”  a  software  tool  somewhat  like  a  “Wizard”  designed  to  help 
inexperienced  Navy  personnel  learn  to  use  a  manpower  planning  tool  (MacMillian  & 
Freeman,  1996).  The  method  allows  an  evaluation  of  the  congruence  of  the  metaphor  and 
the  users’  mental  models  that  is  independent  of  the  evaluation  of  the  value  of  the 
metaphor  in  improving  user  understanding  or  performance. 

Domain 

The  Navy  faces  a  complex  manpower  planning  problem  in  keeping  its  many 
technical  specialty  areas  staffed  at  desired  levels.  The  number  of  new  recruits  that  should 
be  enrolled  for  training  in  each  specialty  every  year  (entry-level  training  is  provided  by 
“A”  Schools)  depends  on  a  number  of  interacting  factors.  These  factors  include:  (1)  the 
number  of  specialists  needed  each  year,  (2)  the  number  of  specialists  developed  through 
on-the-job  training,  (3)  the  expected  rates  at  which  members  of  the  specialty  will  leave 
the  Navy,  (4)  the  dropout  rate  for  the  training  schools,  and  (5)  the  length  of  training 
schools.  The  Navy  Personnel  Research  and  Development  Center  (NPRDC)  has 
developed  a  manpower  planning  tool  to  help  Navy  planners  develop  “A”  School  training 
plans.  This  tool  uses  a  complex  mathematical  model  to  predict  the  required  number  of 
trainees  based  on  input  from  the  user.  Users  new  to  the  tool  often  have  trouble 
understanding  how  to  use  it  and  in  interpreting  its  results. 
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Because  of  the  difficulties  encountered  in  learning  to  use  the  planning  tool,  NPRDC 
developed  a  User  Coach  to  help  inexperienced  users.  The  Coach  provides  step-by-step 
procedural  instructions,  text  explanations,  and  a  pictorial  representation  of  the  factors 
that  affect  “A”  School  enrollment  requirements,  shown  in  Figure  2.  This  pictorial 
representation  uses  a  hydraulic  flow  metaphor— a  series  of  tanks,  pipes,  funnels,  and 
faucets — to  show  users  how  various  factors  affect  the  needed  school  inputs.  Each 
planning  factor  is  represented  by  a  hydraulic  element.  For  example,  water  in  a  tank  is 
used  to  show  the  “inventory”  (the  number  of  individuals  in  a  technical  specialty)  in  each 
year.  The  tank  has  a  desired  fill  line  showing  the  number  of  individuals  needed  in  that 
specialty.  The  pictorial  representation,  called  “The  Big  Picture,”  was  provided  in  an 
attempt  to  improve  users'  conceptual  understanding  of  the  complex  interrelationships 
among  the  factors  affecting  school  enrollment  requirements. 


Figure  2.  The  Big  Picture. 


The  hydraulic  flow  metaphor  was  developed  based  on  interviews  with  users  of  the 
planning  tool  in  which  they  described  their  mental  models  for  manpower  planning 
(Tatum,  1994).  The  interviews  asked  the  users  to  describe  an  analogy  or  metaphor  that 
they  use  to  help  them  do  their  job  and  use  the  planning  tool.  Over  70  percent  of  those 
interviewed  reported  using  a  hydraulic  (fluid)  or  pneumatic  (air  pressure)  metaphor  to 
describe  their  “A”  School  planning  activities  and  use  of  the  software  tool. 

Approach 

We  evaluated  the  effectiveness  of  the  User  Coach  by  having  users  develop  plans  with 
and  without  the  Coach,  and  with  and  without  the  Big  Picture.  As  part  of  this  evaluation, 
we  asked  users  to  complete  a  Metaphor  Rating  Matrix  designed  to  collect  data  that  would 
allow  us  to  analyze  the  congruence  between  users’  mental  models  of  the  “A”  School 
planning  process  and  the  visual  metaphor  used  in  the  Big  Picture. 
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The  rating  matrix,  shown  in  Figure  3,  measures  the  correspondence  between  the 
properties  of  the  various  hydraulic  elements  shown  in  the  picture  and  the  properties  of 
the  various  planning  factors  as  they  exist  in  the  user’s  mental  model  of  planning.  The 
pictorial  elements  in  the  Big  Picture  were  chosen  because  they  had  properties  that 
corresponded  to  the  properties  of  the  concepts  that  they  represented.  For  example,  a 
dripping  faucet  was  used  to  show  attrition  from  the  Navy — individuals  “leak  out”  over 
time.  We  developed  a  list  of  the  properties  that  we  believed  were  shared  by  the  hydraulic 
system  elements  and  the  “A”  School  planning  concepts.  These  are  the  properties 
underlying  the  choice  of  the  hydraulic  metaphor — the  aspects  of  the  metaphor  believed  to 
be  parallel  to  the  user’s  mental  model.  These  properties  formed  the  columns  of  the 
matrix.  The  rows  were  formed  by  the  hydraulic  elements  shown  in  the  Big  Picture,  e.g., 
“faucet”  or  “tank”  and  by  the  abstract  concepts  used  in  planning,  e.g.,  “inventory.”  We 
asked  each  subject  to  rate  the  applicability  of  each  of  the  properties  to  each  of  the 
hydraulic  elements  and  to  each  of  the  planning  concepts.  The  correspondence  between 
subjects’  ratings  of  the  properties  for  a  hydraulic  element  and  their  ratings  for  the 
concept  represented  by  that  element  indicates  how  well  each  concept  was  represented  by 
the  corresponding  pictorial  element. 


Metaphor  Score  Sheet  Subject: _  Session  Code: _ Date: _ 

In  the  table  below,  please  rate  how  well  each  of  the  phrases  at  the  top  of  the  table  describes  die  components  or  concepts  listed  at  the  left. 
These  components  or  concepts  involve:  1)  the  flow  of  water  in  a  hydraulic  system,  or  2)  the  "flow"  of  enlisted  personnel  through  the  A- 
School  and  into  the  inventory 


Provide  your  ratings  by  filling  in  each  open  (white)  cell  below  with  a  number  from  1  to  5 ,  where :  1  =not  at  all  applicable 

5=very  applicable 


Ho  v  well  does  each  phrase  to 
the  right — > 
describe  the  components  or 
concepts  below? 

Involves 

move¬ 

ment 

Involves 

one-way 

move¬ 

ment 

Has  a 

holding 

capacity 

Has  a 

flow 

capacity 

Has  a 
fill  level 

Has  a 

flow 

rate 

Involves 
input 
into  a 
system 

Involves 
exit  out 
of  a 
system 

Con¬ 
nects 
parts  of 
a  system 

Components  of  a 
hydraulic  system 

Faucet 

Water  through  a  faucet 

Tank 

Water  in  a  tank 

Funnel 

Water  through  a  funnel 

Pipe 

Water  through  a  pipe 

A- School  planning 
concepts 

Loss  rates  (from  inventory) 

A- School  attrition 

Non-school  gains 

Inventory  1 

A- School  capacity 

A- School  inputs  (accessions) 

Figure  3.  Metaphor  Rating  Sheet. 
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Results 


Twenty-one  subjects  completed  the  task  of  rating  the  applicability  of  each  of  nine 
properties  for  each  of  eight  different  hydraulic  system  components  and  six  different  “A” 
School  planning  concepts.  Subjects  rated  each  property  on  a  5-point  scale  where  “1” 
indicated  that  the  property  was  not  at  all  applicable  and  “5”  indicated  that  it  was  very 
applicable. 

Mean  Property  Ratings 

The  mean  rating  of  each  property  for  each  component  or  concept,  averaged  over 
subjects,  is  shown  in  Table  1.  Properties  with  high  mean  applicability  ratings  (greater 
than  or  equal  to  a  threshold  value  of  4.0)1  are  shown  in  bold  in  the  table.  For  example, 
the  faucet  component  was  judged  by  subjects  to  have  two  highly  applicable  properties 
among  the  nine:  “has  a  flow  capacity”  and  “involves  exit  out  of  a  system”  with  mean 
ratings  of  4.05  and  4.19,  respectively.  The  number  of  properties  judged  to  be  highly 
applicable  ranged  from  a  minimum  of  one  (water  in  a  tank,  funnel,  and  inventory)  to  a 
maximum  of  four  (water  through  a  faucet  and  loss  rate)  across  the  eight  hydraulic 
components  and  six  planning  concepts. 


Table  1 


Mean  rating  of  Property  Relevance  to  Hydraulic  System  Components 
and  “A”  School  Planning  Concepts 


Involves 

Movement 

Involves 

One-way 

Movemet 

Has  a 
Holding 
Capacity 

Has  a 
Flow 
Capacity 

Has  a 
Fill  Level 

Has  a 
Flow  Rate 

Involves 
Input  Into 
a  System 

Involves 
Exit  Out  of 
a  System 

Connects 
Parts  of 
a  System 

Components  of  a  Hydraulic 
System 

Faucet 

2.90 

3.67 

1.48 

4.05 

1.14 

3.29 

3.19 

4.19 

1.62 

Water  through  a  faucet 

4.52 

4.52 

1.29 

3.19 

1.33 

4.86 

3.19 

4.52 

1.38 

Tank 

1.24 

1.24 

5.00 

1.29 

4.95 

1.62 

1.38 

1.33 

2.19 

Water  in  a  tank 

2.05 

1.43 

2.57 

1.86 

4.00 

1.57 

1.52 

1.38 

1.67 

Funnel 

2.90 

3.86 

1.71 

3.67 

1.76 

2.86 

434 

2.24 

2.71 

Water  through  a  funnel 

4.14 

4.48 

1.29 

2.67 

1.33 

4.67 

3.81 

2.76 

1.86 

2.71 

2.10 

2.52 

4.14 

1.43 

3.19 

3.10 

3.14 

4.76 

4.71 

3.24 

133 

3.05 

1.81 

4.57 

3.57 

3.71 

2.81 

“A”  School  Planning 
Concepts 

Loss  rates 

4.05 

4.19 

1.14 

2.81 

1.57 

4.05 

1.67 

4.86 

1.52 

“A”  School  attrition 

4.05 

438 

1.10 

2.86 

1.38 

3.86 

1.48 

4.86 

1.71 

4.05 

433 

1.10 

3.10 

1.57 

3.71  I 

4.90 

1.67 

1.76 

Inventory 

2.00 

1.19 

3.71 

1.90 

4.10 

1.86 

2.00 

2.05 

1.86 

“A”  School  capacity 

1.71 

1.48 

4.52 

2.57 

4.10 

1.76 

2.81 

1.29 

2.05 

“A”  School  inputs 

4.00 

3.95 

2.05 

3.00 

2.05 

3.43 

4.86 

1.71 

1.52 

Users  provided  ratings  on  a  scale  from  1  to  5.  On  this  scale,  3  is  the  midpoint.  We  would  not  expect 
the  mean  ratings  to  be  at  the  highest  value  of  5,  so  4  seemed  to  be  a  reasonable  threshold  for  a  “highly 
applicable”  rating. 
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Correlation  Between  Hydraulic  System  Components  and  “A”  School  Planning 
Concepts 

We  performed  several  different  analyses  to  determine  the  correspondence  between 
the  pattern  of  mean  property  ratings  for  each  hydraulic  system  component  relative  to  the 
pattern  of  mean  property  ratings  for  each  “A”  School  planning  concept.  In  a  first 
analysis,  we  calculated  the  Pearson’s  correlation  coefficient  between  the  vector  of  mean 
property  values  for  each  hydraulic  system  component  and  the  vector  of  mean  property 
values  for  each  “A”  School  planning  concept.  The  results  of  this  analysis  are  shown  in 
Table  2.  For  example,  the  mean  property  ratings  for  faucet  (2.90,  3.67,  1.48,  etc.)  were 
correlated  with  the  mean  property  rating  for  loss  rate  (4.05,  4.19,  1.14,  etc.),  which 
yielded  the  r  =  .77  in  Table  2.  Correlation  coefficients  that  are  significant  with  a 
probability  of  less  than  or  equal  to  0.05  of  occurring  by  chance  are  indicated  by  one  or 
more  asterisks.  Correlations  that  are  significant  with  a  probability  less  than  or  equal  to 
0.01  are  shown  also  in  bold  type. 


Table  2 

Pearson's  Correlation  Coefficients  Between  Components  of  a  Hydraulic  System 
and  “A”  School  Planning  Concepts 


“A”  School  Planning  Concepts  I 

“A” 

Non- 

“A” 

“A”  School 

Loss 

School 

school 

School 

Inputs 

Rates 

Attrition 

Gains 

Inventory 

Capacity 

Components  of  a 
Hydraulic  System 

Faucet 

0.77 

0.77 

0.54 

-0.74 

-0.72 

0.43 

* 

* 

* 

* 

Water  through  a  faucet 

0.92 

0.90 

0.65 

-0.71 

-0.80 

0.55 

*** 

*** 

* 

** 

Tank 

-0.67 

-0.69 

-0.70 

0.95 

0.89 

-0.52 

* 

* 

* 

*** 

*** 

Water  in  a  tank 

-0.52 

-0.55 

-0.49 

0.90 

0.77 

-0.35 

*** 

* 

Funnel 

0.20 

0.21 

0.87 

-0.76 

-0.46 

0.79 

*** 

* 

* 

Water  through  a  funnel 

0.69 

0.67 

0.88 

-0.76 

-0.71 

0:80 

* 

* 

*** 

* 

* 

** 

Pipe 

-0.06 

-0.02 

-0.03 

-0.50 

-0.36 

-0.18 

Water  through  a  pipe 

0.74 

0.72 

0.68 

-0.73 

-0.83 

0.55 

* 

* 

* 

* 

** 

I  1  Component-concept  match  used  in  metaphor. 
*  p  <  0.05. 


**  p  <  0.01. 

***  p  <  0.005. 
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Number  of  Property  Matches  Between  Hydraulic  System  Components  and  “A” 
School  Planning  Concepts 

Another  way  to  examine  the  correspondence  between  the  pattern  of  property  ratings 
for  hydraulic  system  components  and  “A”  School  planning  concepts  is  by  counting  up 
the  number  of  properties  for  a  pair  that  are  in  agreement.  We  used  a  mean  property  rating 
of  4.0  as  a  threshold  for  determining  whether  a  given  property  was  judged  to  be 
applicable  to  a  component  or  concept.  Thus,  if  a  property  had  a  mean  applicability  rating 
below  4.0,  it  was  assigned  a  “not  applicable”  label;  if  its  mean  rating  was  4.0  or  above,  it 
was  assigned  an  applicable”  label.  Then,  for  each  hydraulic  component,  planning 
concept  pair,  we  counted  the  number  of  properties  that  were  in  agreement— either  both 
“applicable”  or  both  “not  applicable.”  Since  there  were  nine  properties  altogether,  the 
maximum  property  match  score  is  nine  and  the  minimum  is  zero.  The  results  are  shown 
in  Table  3.  For  each  “A”  School  concept,  arrayed  horizontally,  we  have  bolded  and 
underlined  the  score(s)  for  the  hydraulic  system  component(s)  that  received  the  highest 
number  of  property  matches. 


Table  3 


Number  of  Property  Matches  (of  nine  possible)  Between  Components  of  a 
Hydraulic  System  and  “A”  School  Planning  Concepts 


“A”  School  Planning  Concepts  1 

Loss 

Rates 

“A” 

School 

Attrition 

Non¬ 

school 

Gains 

Inventory 

“A” 

School 

Capacity 

“A” 

School 

Inputs 

Components  of  a 
Hydraulic  System 

Faucet 

5 

6 

4 

6 

5 

5 

Water  through  a  faucet 

9 

8 

6 

4 

3 

5 

Tank 

3 

4 

4 

8 

9 

5 

Water  in  a  tank 

4 

5 

5 

9 

8 

6 

Funnel 

4 

5 

7 

7 

6 

8 

Water  through  a  funnel 

7 

7 

5 

4 

6 

Pipe 

8 

4 

4 

6 

5 

5 

Water  through  a  pipe 

7 

6 

6 

6 

5 

7 

I  I  Component-concept  match  used  in  metaphor. 
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Discussion 


Adequacy  of  the  Property  Set 

The  set  of  properties  rated  by  the  subjects  for  their  applicability  to  each  hydraulic 
system  component  and  “A”  School  planning  concept  appear  to  have  been  adequate  as  a  t 

descriptor  set  in  the  following  ways.  Scanning  the  mean  property  ratings  shown  in  Table 
1 ,  one  can  see  that  each  of  the  nine  properties  was  rated  as  highly  applicable  to  at  least 
one  of  the  components  or  concepts,  and  also  as  highly  inapplicable  (a  mean  rating  less 
than  2.0)  to  at  least  one  other  component  or  concept.  Thus,  each  property  was 
discriminating  across  the  set  of  components  and  concepts.  Furthermore,  for  each 
component  and  concept,  there  was  at  least  one  property  that  was  judged  to  be  highly 
applicable  and  at  least  one  other  that  was  judged  to  be  highly  inapplicable. 

Correspondence  Between  Hydraulic  System  Components  and  “A”  School  Planning 
Concepts 

If  we  focus  our  attention  on  only  those  hydraulic  components  and  planning  concept 
pairs  that  are  highly  correlated  (those  shown  in  bold  in  Table  2),  then  we  see  that  there  is 
at  least  one  hydraulic  system  component  that  corresponds  well  as  a  metaphor  for  each  of 
the  “A”  School  planning  concepts,  as  indicated  by  a  high  positive  correlation.  Only  one 
of  the  hydraulic  system  components — pipe — fails  to  be  highly  correlated  with  any  of  the 
“A”  School  planning  concepts. 

Interestingly,  there  is  a  failure  by  the  subjects  to  discriminate  between  a  tank  and 
water  in  a  tank  as  metaphors  for  the  “A  ”  School  inventory.  Both  are  highly  correlated 
with  inventory ,  suggesting  that  subjects  may  have  thought  of  a  tank  not  just  as  the 
container  capable  of  holding  water,  but  as  the  vessel  and  contained  water  combined. 

Similarly,  both  funnel  and  water  through  a  funnel  were  equally  highly  correlated  with 
non-school  gains,  suggesting  that  the  funnel  was  regarded  as  including  the  fluid  that 
flowed  through  it. 

Not  every  planning  concept  had  a  unique  corresponding  component  in  the  hydraulic 
system.  For  example,  both  loss  rates  and  “A”  School  attrition  had  water  through  a  faucet 
as  the  most  highly  correlated  hydraulic  system  component. 

Similar  conclusions  may  be  drawn  generally  from  the  counts  of  property  matches 
shown  in  Table  3.  The  hydraulic  system  component  that  represents  the  best  match  with 
each  “A”  School  planning  concept,  as  determined  by  the  number  of  matching  properties, 
is  the  same  as  determined  by  the  correlation  coefficients,  with  only  two  minor 
exceptions.  The  best  match  for  inventory  in  terms  of  counts  is  water  in  a  tank  rather  than 
tank,  and  the  best  match  for  “A”  School  inputs  is  funnel  rather  than  water  through  a 
funnel. 
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Conclusions 


Overall,  the  results  show  a  high  degree  of  correspondence  between  the  properties  of 
the  A  School  planning  concepts  and  the  properties  of  the  hydraulic  components  that 
were  chosen  to  represent  them  in  the  visual  metaphor.  Water  through  a  faucet  was  used 
to  represent  both  “A”  School  attrition  and  loss  rates;  a  tank  represented  the  capacity  of 
the  “A”  School;  water  in  the  tank  represented  inventory;  and  water  through  a  funnel 
represented  non-school  gains  and  “A”  School  inputs.  In  each  case,  given  the  set  of 
hydraulic  components  with  which  we  were  working,  we  seem  to  have  chosen  the  best 
possible  match  between  components  and  concepts.  We  conclude  that  our  metaphor  was  a 
good  one,  i.e.,  corresponded  well  to  users’  conceptual  models  of  “A”  School  planning. 
Our  results  do  not  necessarily  indicate  that  another  completely  different  metaphor  might 
not  have  been  better,  but  the  measurement  method  that  we  developed  could  be  used  to 
assess  other  metaphors  in  order  to  compare  their  relative  “fit”  with  the  users’  conceptual 
models. 

Our  measurement  method  allows  us  to  differentiate  between  the  “quality”  of  the 
metaphor  used,  measured  in  terms  of  its  fit  to  users’  conceptual  models,  and  the 
usefulness  of  the  metaphor  in  the  interface.  Interestingly,  the  evaluation  results 
(MacMillian  and  Freeman,  1996;  MacMillian,  Freeman,  Tatum,  &  Ropp,  1997)  indicate 
that  the  conceptual  use  of  the  visual  metaphor  in  the  User  Coach  was  not  especially 
helpful  in  increasing  new  users’  understanding  of  “A”  School  planning  concepts.  The 
mean  score  on  a  “concepts  quiz”  that  tested  users’  understanding  of  the  relationships 
among  planning  concepts  was  not  significantly  higher  for  users  who  were  shown  the 
metaphor  (12.3  out  of  a  maximum  score  of  15)  than  for  users  who  did  not  see  the 
metaphor  (10.9  out  of  15).  Also,  the  mean  time  to  complete  the  first  planning  task  was 
17.1  minutes  for  users  who  viewed  the  Big  Picture,  and  13.5  minutes  for  users  who  did 
not.  The  extra  time  required  to  study  the  pictorial  metaphor  did  not  produce  a  measurable 
increase  in  conceptual  understanding. 

We  conclude  that  the  metaphor  was  a  good  one,  but  that  its  use  in  the  Coach— in  an 
attempt  to  increase  conceptual  learning — was  unsuccessful.  Perhaps  metaphors  are  not  as 
helpful  for  conceptual  use  as  for  functional  use  in  interface  designs.  We  have  some 
confidence,  however,  based  on  our  evaluation,  that  a  better  metaphor  would  not  have 
improved  understanding.  Although  exposure  to  the  metaphor  as  implemented  did  not 
improve  users’  conceptual  understanding,  our  results  demonstrate  that  the  metaphor  was 
congruent  with  the  users’  mental  models. 
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